Concerning the problem of wide applications and low efficiency of two-person interaction behavior recognition, a method of two-person interaction behavior recognition based on key gestures was proposed. First, the key frames were extracted by comparing the differences between frames. Second, the key gestures in the key frames were determined by using the variance and spatial relationship of the angle changes of the bone points. Then, the key gestures were represented by features such as joint distance, angle, and joint motion. Every key gesture was expressed as a feature matrix. Finally, the combination with the best recognition rate was selected by comparing different dimension reductions and classification combinations. The proposed recognition method was evaluated on the SBU interaction dataset and the self-built interaction dataset, and the recognition rate of it reached 92.47% and 94.14% respectively. Experimental results show that the proposed method of representing actions by extracting the features of key gestures to form feature matrices can effectively improve the recognition result of two-person interaction behavior.